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ABSTRACT 



A structure guided image processing system uses geometric 
structure information to guide image feature extraction and 
enhancement of an input image to produce a weight image 
output and a mask image output. Geometric structure infor- 
mation may be apparent in the nature of the images, or it can 
in many cases be derived from CAD information. Idempo- 
tent processing and filtering operations minimize image 
distortion. Directional elongated structuring elements pro- 
vide structure-guided selective processing and high speed 
filtering throughput. 

7 Claims, 12 Drawing Sheets 
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STRUCTURE-GUIDED IMAGE PROCESSING 
AND IMAGE FEATURE ENHANCEMENT 
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TECHNICAL FIELD 

This invention relates to image processing methods that 
incorporate knowledge of object structure derived from the 
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image itself or from a-priori knowledge of an object's 
structural relationships from its design data (such as CAD 
drawings) to enhance object features and/or guide image 
measurement estimation and object detection. 

s 

BACKGROUND OF THE INVENTION 

Common tasks in computer vision applications include 
enhancement and detection of objects of interest, refinement 
of detected object masks, and measurement, alignment or 

10 classification of the refined object. Other applications 
include enhancement for image compression or image high- 
lighting for display. Many computer vision applications 
require the enhancement and measurement of image features 
for objects of interest characterization or detection. Appli- 

15 cation domain knowledge is available in most computer 
vision applications. The application domain knowledge can 
often be expressed as structures of image features such as 
shaped color, edges, fines and regions, or changes with time 
such as object motion on a prescribed path. The structures 

20 include spatial relationships of object features such as shape, 
size, intensity distribution, parallelism, co -linearity, 
adjacency, position, etc. The structure information can be 
particularly well defined in industrial applications such as 
semiconductor, electronic or machine part inspections. In 

25 machine part inspections, most of the work-pieces have 
available Computer Aided Design (CAD) data that specifies 
CAD components as entities (e.g. LINE, POINT, 3DFACE, 
3DPOLYINE, 3D VERTEX, LINE, POINT, 3DFACE, 
3DPOLYUNE, 3DVERTEX, etc.) and blocks (properties 

30 that are associated) of entities. Semiconductor applications 
frequently have step and repeat type processes that form 
lines, patterns, and mosaic structures. In biomedical or 
scientific applications, structure information may also be 
loosely defined. For example, a cell nucleus is generally 

35 round, frequently stains dark, and different but known 
approximate shapes can differentiate different types of blood 
cells or chromosomes. 
The capability of a computer vision system is often 

40 characterized by its detection/measurement accuracy, 
repeatability and throughput. It is desirable to achieve sub- 
pixel measurement accuracy and repeatability for many 
computer vision applications. Application domain knowl- 
edge used according to this invention can significantly 

45 improve the capability of a computer vision system to make 
accurate and repeatable measurements. However, it is non- 
trivial to efficiently use the application domain knowledge in 
high precision applications. 

PRIOR ART 

50 

Prior art uses an image segmentation approach for image 
feature detection or measurement (Haralick R M and 
Shapiro, L G, "Survey Image Segmentation Techniques", 
Comput. Vision, Graphics, and Image Processing, vol. 29 

55 No. 1: 100-132, January 1985). The image segmentation 
approach converts a grayscale image into a binary image 
that contains object of interest masks. Binary thresholding is 
a common technique used in the image segmentation 
approach to create masks. 

60 Because edges or features of an image are imaged by the 
optical and imaging system as continuously varying gray 
levels, there exists no single gray level that represents edge 
pixels. For this reason, any system that depends on taking a 
binary threshold of the image before critical dimensions are 

65 determined must necessarily introduce quantization errors 
into the measurement. Binary thresholding also exacerbates 
the resolution limiting effect of system noise. 
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Prior art applies application domain structure information structure guided image feature enhanced image. The weight 

through a projection/dispersion approach. The projection/ image output and/or the mask image outputs can be used for 

dispersion approach integrates image pixel values in a image compression, highlighting and display of an image, 

pre-defined direction in the image. This can be done using a measurement of objects within the feature enhanced image, 

binary image (projection) or grayscale image (dispersion) 5 or object detection. 

and results in a one-dimensional plot of summed pixel SUMMARY OF THE INVENTION 
values. The application domain structure information 0i . , , , , . , 
defines the projection directions, however misalignments, Stru ?. ure ^ mo JP holo P ad ^ use . s a "P no " 
variations in illumination, and image noise limit the resolv- «f omettlc "V"*™ «° °J op*™* «» 
ing capability of these projections. Hie prior art approach is 10 «™f»™« el f me f o extract un age features of par- 
sensitive to system variations such as rotation, object mterest to ** module - Structure-gurded morphologi- 
illumination, changes in object surface texture (which f P™ 6 * 311 ? u,volv6S ■ of morphological opera- 
affects gray levels), etc. Rotation errors result in the inte- Uons Y«>™» »» ™& *•!>«« e .l eme f> **. 
gration of pixel values along a wrong direction that is onc ? ( . to , the , h & h f l s P 6Clfic sha P e ° r 
destructive to accuracy. Furthermore, the projection-based is P os ™ /T*, • u ™ £ - P "Bpo'tance to the 
approach cannot effectively combine multiple two- Unction of the algoritlirn. This mvention seeks to provide 
dimensional structure information (such as anti-parallelism, high performance image feature extraction and enhancement 
orthogonality, intersection, curvaceous qualities) where fea- a ^ture-guided image processing method, 
tares of interest may be along different directions or com- BRIEF DESCRIPTION OF THE DRAWINGS 
plex. Another difficulty in the prior art is that two- 20 FIG. 1 shows a structure-guided image measurement 
dimensional processing is needed for reliable sub-pixel system 

accuracy due to the utility of using as many pixels as FIG ' 2 shows a steuctule . gllilied detection system . 

possible tor the measurement. Use or all possible pixels ™^ ~ . u ■ i_« j t 

... 4 . , , \ -j FIG. 3 shows bright edge extraction by grayscale mor- 

mimmizes spatial quantization errors and also aids recon- , , . t -j 

j • * 1 * • i_ 4 , ! ri . phological erosion residue, 

struction and interpolation between sample values. Herein 25 M * * , , . 

there are two difficulties, the prior art does not take advan- HG - 3A shows an m P ut ram P ed & e ima S e: 1 

tage of all pixels whose position is related, the prior art FIG ' 3B shows structurmg element: A 

confuses image surface information and image edge infor- FIG. 3C shows eroded image: 16 A 

mation through the use of projection, and the projections FIG. 3D shows erosion residue: I-I0A 

cannot be used effectively with complex structures. Where 30 FIG. 4 shows dark edge extraction by grayscale morpho- 

the prior art could have employed two dimensions to achieve logical dilation residue. 

a better result (but not a projection result), such grayscale FIG. 5 shows general edge extraction by difference of 

processing is in the prior art computationally demanding and grayscale dilation and erosion. 

requires expensive and extensive special hardware to FIG. 6 shows an example of line/region detection and 

achieve desired throughput. Additionally, in an effort to 35 contrast extraction. 

enhance image features and thereby improve measurement FIG. 7 shows examples of structure-guided general edge 

signal to noise or object detection or classification accuracy, extraction that extracts vertical edges and horizontal edges 

prior art uses linear filters. Linear filters are derived from a using different directional elongated structuring elements, 

digital signal processing paradigm where structure informa- FIG. 8 shows examples of structure-guided dark line 

turn is considerably less obvious. Unear filters are not *o extractkm ^ extract& horizontal ^ ^ vertical ^ 

designed to input structure information and therefore cannot usi agonal elongated structuring elements 

utilize application domain structure knowledge. Where lin- n A , . , . 

n U , . * • ,i . t c c < FIG. 9A shows an input noisy edge intensity profile 

ear filters have been used in the pnor art for feature n _ . , * c 

enhancement, their own characteristics obscure essential ™- *B shows the -result of opening the noisy input image 

image characteristics because they introduce phase delay 45 using structuring element A 

distortion that causes image blur, under-shoot, over-shoot or FIG * 9C shows the result of a closm S operation using 

ringing and edge displacement. These image distortions structuring element A on the result shown in FIG. 9B to 

increase uncertainty of feature measurement. Image vari- produce a new result. 

ability and noise in conjunction with prior art linear filtering FIG. 9D shows the result of another opening operation 

and thresholding seriously degrade measurement reliability 50 usin S structuring element B on the result shown in FIG. 9C 

and accuracy. to produce a new result. 

FIG. 9E shows the result of a closing operation using 

OBJECTS AND ADVANTAGES structuring element B on the result shown in FIG. 9D to 

It is an object of this invention to provide improved image produce the final result 

feature extraction, and feature enhancement through a 55 FIG. 10 shows the mask generation processing steps 

structure-guided image processing method. It is another FIG. UA shows the same input noisy edge intensity 

object of the invention to enhance image features through profile as FIG. 9A 

use of nonlinear image processing that does not introduce FIG. 11B shows the result of opening the noisy input 

phase shift and/or blurry effect (transient aberration). A image of 11 A using structuring element B 

further object of the invention is to provide methods for 60 FIG. 11C shows the result of a closing operation using 

utilizing application domain knowledge encoded into the stnict uring element B on the result shown in FIG. 11B to 

image processing parameters for structure -guided extraction produce a new result 

and enhancement of features of interest and/or to remove 

noisy or irrelevant information. It is a further object of the DETAILED DESCRIPTION OF THE 

invention to create an object mask image output from the 65 INVENTION 

structure guided image feature enhanced image. A further This invention provides sub-pixel, high performance 

object is to provide a weight image output derived from the image feature extraction and enhancement through a 
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structure-guided image processing method. The processing 
involves two-dimensional, full grayscale processing that can 
be implemented efficiently and cost-effectively. The image 
processing does not introduce phase shift and/or blurry 
effect. In a structure-guided image processing method, appli- 5 
cation domain knowledge is encoded into the parameters for 
structure-guided extraction and enhancement of features of 
interest and removal of noisy and irrelevant information. 

I. Structure-guided Image Processing System 10 

FIG. 1 and FIG. 2 show the processing flows of two 
application scenarios of this invention. FIG. 1 shows a 
structure-guided image mask generation and image weight 
generation system and FIG. 2 shows applications for the 
outputs of the system described in FIG. 1. 

As shown in FIGS. 1 and 2, the input image 100, 200 is 
processed by a structure-guided image feature extraction 
module 102, 202 to extract image features of interest 104, 
204. The image feature extraction module may not be ^ 
needed if the desired image features are already presented in 
the input image. A structure-guided image feature enhance- 
ment module 106, 206 enhances the image features. The 
feature-enhanced image is the input for a mask generation 
module 110, 208 and a weight generation module 120, 216. 
The mask generation module 110, 208 generates masks 
containing features of interest 112 and uniquely labels each 
connected component of the masks for follow-on 
processing, image compression or display/highlight. The 
weight generation module generates a grayscale weight 3Q 
image 118,212. The weights correspond to the strength of 
the features of interest in the image. FIG. 2 shows the 
applications that can be accomplished with the image feature 
mask(s) and the weight image. These outputs are used for 
image compression, image display highlighting, image 35 
measurement, and/or feature detection. The detailed descrip- 
tion for an image measurement embodiment is disclosed in 
co -pending U.S. patent application entitled, "Structure- 
guided Image Measurement Method" by Shih-Jong J. Lee et. 
aL, filed Dec. 15, 2000 which is incorporated in its entirety 4Q 
herein. 

II. Structure -guided Image Feature Extraction 

Image features are characterized by their grayscale (or 
color) intensity distributions and their spatial (or temporal) 45 
structures. Major image features include linear features such 
as bright edge, dark edge, general edge, bright line, dark line 
and general line. Major image features also include image 
regions such as bright region, dark region and general 
region, etc. Linear features can be arranged vertically, 50 
horizontally, and/or in different spatial directions. Regions 
can exhibit different shapes such as circular or rectangular, 
etc. They can be arranged in a given structure including 
relative locations, orientation or symmetry. Other image 
features include small regions or points such as comers of 55 
regions or intersection points of different features (linear 
features and/or regions). 

The structure-guided image feature extraction system 102 
of this invention efficiently extracts image features of inter- 
est and removes noisy and irrelevant information. In one 60 
embodiment of this invention, this is accomplished by a 
sequence of grayscale morphological processing that 
encodes structure information into directional elongated 
structuring elements that can be efficiently implemented 
using a general purpose computing platform (co-pending 65 
patent applications entitled 'U.S. patent application Ser. No. 
09/693,723, "Image Processing System with Enhanced Pro- 



cessing and Memory Management", by Shih-Jong J. Lee et. 
al, filed Oct. 20, 2000 and U.S. patent application Ser. No. 
09/692,948, "High Speed Image Processing Apparatus 
Using a Cascade of Elongated Filters Programmed in a 
Computer", by Shih-Jong J. Lee et. al., filed Oct. 20, 2000). 
II. 1 Feature Extraction Processing Sequence 

This section describes some feature extraction processing 
sequences in the preferred embodiment of the invention. 

Bright Edge Extraction 

In a preferred embodiment, bright edges are extracted by 
a grayscale erosion residue processing sequence defined as: 

I-I8A 

Where I is an input image, A is a structuring element and 
6 is the grayscale morphological erosion operation 
(Sternberg, S R, "Gray-scale morphology," Comput. Vision, 
Graphics Image Processing, vol. 35: 333-355, 1986), FIGS. 
3A, 3B, 3C, 3D illustrate the grayscale erosion residue 
operation applied to a one dimensional grayscale ramp edge 
I 300 shown in FIG. 3A. FIG. 3B shows the structuring 
element A308 and FIG. 3C shows image 1 300 eroded by A 
308 resulting in eroded image 310. The erosion residue 
result 314 is shown in FIG. 3D. Spatial alignment marks 
302, 304, 306 show the position shifts that occur as a result 
of erosion and erosion residue operations. Notice that the 
original image I has its increase starting at 302 and ending 
at 304. By use of an appropriately sized structuring element 
A shown in 308, an eroded image result 310 is displaced an 
amount equal to the structuring element size. In this example 
1 310 has been displaced and now begins at position 304. An 
erosion residue result: 

1-I8A 

shown as 314 in FIG. 3D, demonstrates that grayscale 
morphological bright edge detection may not introduce 
undesired phase shift, group envelope delay distortion or 
blurry effect (transient aberration) normally caused by linear 
filters and the position and size or shape of image features 
can remain undisturbed, thus enhancing image features. 
Dark Edge Extraction 

In the preferred embodiment, Dark edges are extracted by 
a grayscale dilation residue processing sequence defined as: 

I0A-I 

Where © is the grayscale morphological dilation opera- 
tion (Sternberg, S R, "Gray-scale morphology," Comput. 
Vision, Graphics Image Processing, vol. 35: 333-355, 
1986). FIG. 4 illustrates the grayscale erosion residue opera- 
tion applied to the one-dimensional ramp edge 408. FIG. 4 
shows the result 400 of dilating image I 408 by structuring 
element A 410. The dilation residue result 412 is also shown 
in FIG. 4. As shown in FIG. 4, grayscale morphological dark 
edge detection with a selected structuring element does not 
introduce undesired phase shift or blurry effect (transient 
aberration). 

General Edge Extraction 

General edges (both dark and bright edges) can be 
extracted by the difference of grayscale dilation and erosion 
defined as: 

I0A-ieA 

FIG. 5 illustrates the one dimensional ramp edge I, 512, 
the grayscale dilation result of I by the structuring element 
A 514, 500, and the gray scale erosion of I by A result 510. 
The difference of grayscale dilation and erosion result is 
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516. As shown by 516, grayscale morphological edge detec- 
tion does not introduce undesired phase shift or blurry effect. 
Bright Line/region Extraction: 

In the preferred embodiment, bright lines/regions are 
extracted by a grayscale opening residue processing 5 
sequence defined as: 

I-(IoA) 



10 



where o is the grayscale morphological opening operation. 
FIG. 6 illustrates a grayscale opening residue operation 
applied to a one-dimensional image profile 600. In FIG. 6 
the opening of image I, 600, by a sufficiently large struc- 
turing element produces result 604. The opening residue 
result 608 obtained from the input image gray level profile 
600 demonstrates that grayscale morphological line/region 
detection does not introduce undesired phase shift or blurry 
effect. 

Dark line/region extraction 

Dark lines/regions can be extracted by a grayscale closing 2Q 
residue processing sequence defined as: 

([•A)-I 

where • is the grayscale morphological closing operation. 
In FIG. 6 the closing of image I, 600, by a sufficiently large 25 
structuring element produces result 602. The grayscale mor- 
phological closing residue result 606 obtained from the input 
image gray level profile 600 demonstrates that grayscale 
morphological line/region detection does not introduce 
undesired phase shift or blurry effect. 30 
Region contrast extraction 

In one preferred embodiment of the invention, region 
contrast is extracted by the difference of grayscale closing 
and opening operations on the input image 600. The pro- 
cessing sequence is defined as: 35 

([•A)-(IoA) 

In FIG. 6 a large structuring element A is assumed. A is 
assumed to be larger than the irregularities shown in the ^ 
input image 600 and therefore produces the gray scale 
opening result 604 and the gray scale closing result 602 that 
is illustrated. The difference between these grayscale closing 
and opening results 610, is a measure of regional image 
contrast. A reference for the contrast result 610 is shown in 45 
612. The morphological region contrast extraction result 610 
does not exhibit any undesired phase shift or blurry effect. 

Region Boundary Extraction: 

Since two sides of a line are also edges, edge extraction 
operations will extract lines as well as edges. When using the 50 
same structure element, the edge extraction results, (bright 
edge extraction, dark edge extraction and general edge 
extraction disclosures), include the corresponding line/ 
region extraction results (bright line/region extraction, dark 
line/region extraction, and region contrast extraction 55 
disclosures). A region boundary consists of only edges that 
are in the boundaries of large regions and excludes edges 
from nanow lines. 

In the preferred embodiment, a bright region boundary is 
the difference between grayscale morphological opening 60 
and erosion: 

loA-IBA 

Similarly, a dark region boundary is the difference 
between grayscale morphological dilation and closing: 65 

I©A-I#A 



And a general region boundary is the difference between 
the summation of grayscale morphological opening and 
dilation and the summation of grayscale morphological 
erosion and closing: 

(IoA+I©A)-(ieA+I#A). 

II .2 Structure-guided Feature Extraction Processing 
Sequence 

By chosen the proper structuring element for the feature 
extraction processing sequence, structure-guided feature 
extraction can be efficiently accomplished. In a preferred 
embodiment of this invention, features of different structures 
are extracted using directional elongated structuring ele- 
ments. Directional elongated structuring elements have lim- 
ited width in one of its dimensions. It can be efficiently 
implemented in a general-purpose computer using the meth- 
ods taught in co -pending U.S. Patent Applications entitled 
"U.S. patent application Ser. No. 09/693,723, "Image Pro- 
cessing System with Enhanced Processing and Memory 
Management", by Shih-Jong J. Lee et. al, filed Oct. 20, 2000 
and U.S. patent application Ser. No. 09/692,948, "High 
Speed Image Processing Apparatus Using a Cascade of 
Elongated Filters Programmed in a Computer", by Shih- 
Jong J. Lee et. al., filed Oct. 20, 2000. The direction of the 
elongated structuring element is chosen to be approximately 
orthogonal to the primary direction of the features to be 
extracted. The process works even if the input edge is 
slightly rotated. 

FIG. 7 shows two structure-guided general edge extrac- 
tion examples. One example extracts horizontal edges (co- 
linear with 700 and 720) from an input step edge image 704. 
A second example extracts vertical edges (co-linear with 702 
and 706) from an input step edge image 704 using direc- 
tional elongated structuring elements 708, 714. To extract 
vertical edges a horizontal elongated structuring element 
714 is used. The general edge extraction processing 
sequence (difference between grayscale dilation and 
erosion) is applied and the vertical edges are extracted using 
structuring element 714 to produce result 718, 716. 
Similarly, a vertical elongated structuring element 708 is 
used for horizontal edge extraction to produce result 710, 
712. In these examples, the width of the directional elon- 
gated structuring element (714 or 708) is one pixel and the 
length of the structuring element determines the width of the 
extracted edges 710, 712 and 718, 716. Note that the medial 
axes of the extracted edge lines correspond to the position of 
the input step edge (702, 706 and 700, 720) and no phase 
shift or blurry effect is introduced in the process. 

FIG. 8 shows two examples of the preferred embodiment 
for structure-guided line extraction from an example rect- 
angular line image 804. The first example extracts horizontal 
lines (co-linear with 800, 820) and another example extracts 
vertical lines (co-linear with 802 and 806) using a direc- 
tional elongated structuring element 814, Similarly, a verti- 
cal elongated structuring element 808 is used for horizontal 
line extraction. In these examples, the width of either 
directional elongated structuring element 808, 814 is one 
pixel and the length of the directional elongated structuring 
element is longer than the width of the extracted lines. 
Structure-guided feature extraction allows the separate 
extraction of features of interest defined by their structures 
and irrelevant features are ignored or removed. For example, 
if only the horizontal edges are of interest, they can be easily 
extracted using stnicturing element 708 and no vertical edge 
components are included in the result. 

Those skilled in the art should recognize that the extrac- 
tion of features from any direction can be accomplished with 
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the stnicture-guided feature extraction approach of this 
invention and features extracted from multiple directions 
can be combined by a union (maximum) of multiple direc- 
tional features or intersection (minimum) of different direc- 
tional features (to detect corner points, for example). 5 
Furthermore, two-dimensional structuring elements of dif- 
ferent size and shape can be used to extract desired regions. 
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III, Structure-guided Image Feature Enhancement 

The extracted image features could be noisy and could 
contain irrelevant information. The structure -guided image 
feature enhancement system of this invention efficiently 
enhances image features of interest and removes noisy and 
irrelevant information. In one embodiment of this invention, 
the structure-guided image feature enhancement is accom- 
plished by an increasing idempotent (Serra, J, "Image analy- 
sis and mathematical morphology," London: Academic, 
pp31 8-321, 1982.) processing sequence such as grayscale 
morphological opening and closing operations applied alter- 
natively to image foreground and background. Morphologi- 
cal opening and closing operations possess an increasing 
property that maintains inclusion relationships on the images 
they are applied to. If an inclusion relationship exists for 
structuring elements A and B, that is: 

a<=b, 

then combinations of opening and closing have the follow- 
ing properties: 

30 

(((I»A)oA)#B)oB cz (I*B)oB 
(((IoA)*A)oB)»B =» (IoB)«B 

This means that processing sequences that progressively 
apply combinations of openings and closings are less severe 35 
and introduce less distortion when a small structuring ele- 
ment is used before a larger one. In the preferred 
embodiment, application domain structure information is 
used in selecting the size and shape of structuring elements 
to achieve structure-guided enhancement using an increas- 40 
ing idempotent processing sequence. FIGS. 9 A, 9B, 9C, 9D, 
9E illustrate an indempotent structure-guided feature 
enhancement processing sequence on a noisy edge intensity 
profile 900. Structuring elements A 902 and B 912 chosen 
such that A C B, are used for the processing. FIG. 9B 45 
illustrates the effect of opening using structuring element A 
902 producing result 904. FIG. 9C shows the same struc- 
turing element A 902 used in a closing operation on the 
initial result 904 to produce result 908. FIG. 9D illustrates 
the effect of further opening using structuring element B on 50 
result 908 to produce a new result 914. Finally, the struc- 
turing element B is applied once again for closing 914 to 
produce a result 918 shown in FIG. 9E. FIG. 11 A repeats the 
noisy edge intensity profile 900 as 1100. To illustrate the 
inclusion relationship taught above, the structuring element 55 
B was used to open 1100 to produce result 1102 shown in 
FIG. 11 B. The same structuring element B 1106 was then 
used to open 1102 to produce result 1104 shown in FIG. 11C. 
The feature enhancement process removes noise and pre- 
serves the structure of the features of interest. Using small 60 
structuring elements before larger structuring elements mini- 
mizes distortion. There is little blur, ringing, overshoot or 
pre-shoot normally caused by phase distortion of linear 
filtering. 

By choosing the structuring elements according to the 65 
purposes known for the application, structure-guided feature 
enhancement is accomplished. In one embodiment of this 



invention, features of different structures are enhanced using 
directional elongated structuring elements. Directional elon- 
gated structuring elements are described in a co-pending 
U.S. patent application Ser. No. 09/693,723 entitled, "Image 
Processing System with Enhanced Processing and Memory 
Management", by Shih-Jong J. Lee et. al, filed Oct. 20, 2000 
and U.S. patent application Ser. No. 09/692,948, "High 
Speed Image Processing Apparatus Using a Cascade of 
Elongated Filters Programmed in a Computer*', by Shih- 
Jong J. Lee et. al, filed Oct. 20, 2000 both of which are 
incorporated in their entirety herein. The direction of the 
structuring element is chosen to align with the primary 
direction of the features to be enhanced. The largest size of 
the structuring element in the idempotent processing 
sequence should be smaller than the smallest size of features 
to be enhanced. 

Those skilled in the art should recognize that the 
structure-guided feature enhancement process could start 
with grayscale opening followed by grayscale closing or 
start with grayscale closing followed by opening. Opening 
first will enhance dark features and closing first will enhance 
bright features. Each opening and closing iteration could use 
the same size structuring element for detailed feature refine- 
ment or could use an increased size structuring element for 
more aggressive feature refinement. Elongated structuring 
elements of orthogonal directions could be alternatively or 
sequentially applied in the enhancement processing 
sequence for multiple direction feature enhancement. 

IV. Mask Generation 

In the preferred embodiment the mask generation stage 
110, 208 of structure-guided image processing and image 
feature enhancement generates masks containing features of 
interest and assigns a unique label for each connected 
component of the mask image to be used in follow-on 
grayscale processing (image compression, image highlight 
display, image measurement, and detection 209). For either 
embodiment, the mask generation processing steps are 
shown in FIG. 10. An image thresholding step 1002 is 
applied to the feature enhanced image 1000 to generate 
binary gray level masks of the object regions of interest. If 
the measurement of an object boundary is desired, boundary 
masks are generated 1004. In one embodiment of the 
invention, boundary masks are generated using a general 
edge detection method derived from the difference of binary 
dilation and erosion. A connected component labeling step 
1006 (ref: U.S. patent application Ser. No. 09/702,629, 
"Run-Length Based Image Processing Programmed in a 
Computer", by Shih-Jong J. Lee, filed Oct. 31, 2000) is 
applied to the boundary masks to assign a unique label 1008 
for each connected component of the mask image. 

The purpose of the generated masks is to provide rough 
regions of interest for applying fine grayscale detection or 
measurement. As long as the grayscale images are carefully 
prepared, high accuracy and repeatability of the binary 
masks are not necessary. The image thresholding step can 
therefore use a pre-defined threshold value. For applications 
with significant variations, an adaptive histogram threshold- 
ing method can be used to account for the image-to-image 
variation. In the preferred embodiment the adaptive histo- 
gram thresholding method assumes that an image histogram 
contains a mixture of two Gaussian populations and deter- 
mines the threshold value from the histogram that yields the 
best separation between two populations divided by the 
threshold value (ref.: Otsu N, "A Threshold Selection 
Method for Gray-level Histograms," IEEE Trans. System 
Man and Cybernetics, vol. SMC-9, No. 1, January 1979, pp 
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62-66). Since the feature enhanced image 108 or 207 has 
already removed ambiguous or irrelevant information and 
noise, the mixture of two Gaussian histograms assumption 
yields useful results even when the Gaussian assumption is 
not strictly correct. 5 

V. Weight Generation 

In the preferred embodiment, the weight generation stage 
120, or 216 generates a weight image 118, 212 that forms the 
basis for image compression, image highlight display, gray- 
scale structure-guided estimation and measurement or detec- 
tion 209. The grayscale feature enhanced image 108, 207 
could be used directly as the weight image for the estimation 
if the enhanced features are the subjects of measurement or 
detection. If boundaries of the features are the subjects of 
estimation, a boundary weight image is derived by applying 
the general edge extraction method to the feature enhanced 
image. As this invention teaches, in one embodiment of this 
invention, the difference of grayscale dilation and erosion is 
used for general edge detection. According to the estimation 
or measurement desired, the masked image area is weighted 
with a feature enhanced image. The weights allow gray level 
control of portions of the image according to the importance 
they represent to the estimation or measurement. 

VI. Application Domain Structure Information 

The application domain structure information can be 
derived from Computer Aided Design (CAD) data that 
specifies its components as entities (LINE, POINT, 30 
3DFACE, 3DPOLYLINE, 3DVERTEX, LINE, POINT, 
3DFACE, 3DPOLYLINE, 3D VERTEX, etc.) and blocks 
(related components) of entities. The CAD data can be 
imported from different formats such as IGES, DXF, DMIS, 
NC files, Gerber and Excellon. There are many tools to 35 
automatically display CAD data for easy viewing. 
Alternatively, the application domain structure information 
such as the directions of features and their spatial relation- 
ship such as parallel groups, co-linear groups, intersection 
angles, etc. can be specified by users. 40 

In a preferred embodiment, the structure information is 
derived from CAD data and used to process the input image 
100, 200 through use of a sequence of structuring elements 
of different size and shape according to the methods 
described herein. 45 

The weight image output 118 and mask image output 112 
are particularly useful for image measurement or detection 
as described in the co -pending U.S. Patent Application 
entitled, "Structure-Guided Image Measurement System" by 
Shih-Jong J. Lee et. al., submitted Dec. 15, 2000 which is 
incorporated in its entirety herein. 

The invention has been described herein in considerable 
detail in order to comply with the Patent Statutes and to 
provide those skilled in the art with the information needed S5 
to apply the novel principles and to construct and use such 
specialized components as are required. However, it is to be 
understood that the inventions can be carried out by spe- 
cifically different equipment and devices, and that various 
modifications, both as to the equipment details and operating 6Q 
procedures, can be accomplished without departing from the 
scope of the invention itself. 
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What is claimed is: 

1. A structure-guided image feature enhancement system 
comprising: 

a. an input image; 

b. a plurality of structuring elements selected according to 
feature structures to be enhanced; 

c. order the plurality of structuring elements by increasing 
size; 

d. an increasing idempotent processing sequence operates 
on the input image to enhance features using the 
ordered plurality of structuring elements, 

2. The system of claim 1 wherein the plurality of struc- 
turing elements includes directional elongated structuring 
elements. 

3. The system of claim 1 wherein the idempotent pro- 
cessing sequence includes morphological opening followed 
by closing operations. 

4. The system of claim 1 wherein the idempotent pro- 
cessing sequence includes morphological closing followed 
by opening operations. 

5. A structure -guided image processing system compris- 
ing: 

a. a structure -guided image feature enhancement module 
having an image input and having a feature enhanced 
image output, wherein the feature enhancement module 
uses idempotent processing; 

b. a mask generation module that is connected to the 
enhanced image output having a mask image output; 

c. a weight generation module connected to the structure 
guided image feature enhancement output having a 
weight image output. 

6. A structure -guided image processing system compris- 
ing: 

a. a structure -guided image feature enhancement module 
having an image input and having a feature enhanced 
image output, wherein the feature enhancement module 
uses nonlinear filtering methods; 

b. a mask generation module that is connected to the 
enhanced image output having a mask image output; 

c. a weight generation module connected to the structure 
guided image feature enhancement output having a 
weight image output. 

7. A structure -guided image processing system compris- 
ing: 

a. a structure-guided image feature enhancement module 
having an image input and having a feature enhanced 
image output; 

b. a mask generation module that is connected to the 
enhanced image output having a mask image output, 
wherein the mask generation module further com- 
prises: 

i. an image thresholding module; 

ii. a connected component labeling module 

c. a weight generation module connected to the structure 
guided image feature enhancement output having a 
weight image output. 
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